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(54) COORDINATE SYSTEM SETTING METHOD USING VISUAL SENSOR 

(57)Abstract: 

PURPOSE: To set a coordinate system for 
operation which does not require skillfulness with 
stable precision by observing a coordinate system 
representing means, which visually represents the 
coordinate system, through the visual sensor 
which has a camera means supported on a robot, 
etc. 

CONSTITUTION: An operator operates a robot 
controller to move the robot 1 to a 1 st proper 
« 'olTN. jl'-^ position A 1 where a jig 3 is easily put in the 
y'Cc * visual field of the camera 2, and data on a matrix 

[Al] representing Al are transferred from the 
controller 1 and stored. Then an observation is 
made at the position Al through the camera 2 to 
store the image of a dot group 4 in a frame 
memory, and data of [Dl] are found from pixel 
values are the respective dots in the dot group 4 
and jig data. Similarly, the robot 1 is moved to a 
2nd proper position A2 and data on a matrix [A2] 
are transferred and stored. An observation is made at the position A2 through the camera 
2 and an image of a dot group 4 is inputted to find data of [D2]. Those data are used to 
calculate a matrix [C], thereby completing the setting of the coordinate system. 
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CLAIMS 



[Claim(s)] 

[Claim 1]A coordinate-system-setting method characterized by comprising the following 

using a visual sensor. 

(a) It is made to take several positions which make an automaton with which at least one 
standard coordinates are set up support a camera means of a visual sensor, and are 
different, A stage of observing a coordinate system expressive medium which expresses a 
coordinate system wishing setting out visually in each of a position of these plurality by 
said camera means, and gaining sensor data about said coordinate system wishing setting 
out. 

(b) A stage of gaining data showing a position and a posture of said coordinate system 
wishing setting out to said at least one standard coordinates based on data showing said 
several mutually different positions of an automaton, and said gained sensor data. 



DETAILED DESCRIPTION 



[Detailed Description of the Invention] 
[0001] 

[Industrial Application]Concerning the method of setting a coordinate system as 
automata, such as a robot, in more detail, a visual sensor is used in this invention. 
Therefore, it is related with the method of setting a coordinate system as coordinate 
system expressive media, such as a jig for coordinate system setting, by non-contact at 
automata, such as a robot. 

[0002] 

[Description of the Prior Art] In the automaton of the mold provided with the movable 
part which performs route movement according to the operation program which contains 
the data (only henceforth "position data") showing a position and a posture like a robot 
and a machine tool, two or more coordinate systems are set up and the conformity 
coordinate system at the time of operation is chosen according to work contents. 
[0003]If the standard coordinates (henceforth a base coordinate system) fixed on the base 
of an automaton are ending with setting out, setting out of other coordinate systems will 
be attained by defining the transformation-matrix data showing the position and posture 
over the base coordinate system set [ this ] up. 

[0004]In setting up a new coordinate system with such a method conventionally, Carry 
out fixed installation of the jig which expressed the coordinate system by the point group 
etc. to the position wishing setting out, and two or more prescribed spots of this jig are 

tagged up one by one in the portion representing the position of automata, such as a 
robot, Asking for transformation-matrix data is performed from the data of the touch 
position on a jig, and the position data of the automaton at the time of touch-up. 
[0005]The point of specifically expressing the starting point as the 1st point, for example 
(0, 0, 0), The point of expressing X shaft orientation as the 2nd point (xO, 0, 0); Point (0, 
yO, 0);yO !=0 on xO !=0 and the Y-axis which expresses the direction of an XY plane as 
the 3rd point is chosen. The pin which an automaton tends to touch is set to these three 
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points, manual operation (jogging) of the automaton is carried out, and these pins are 
touched directly [ order ]. And the data (transformation-matrix data) which expresses 
relative position and posture over the base coordinate system of the coordinate system 
expressed with the jig from the position data of the automaton at the time of each tatting 
and the data of a touch position is called for. 
[0006] 

[Problem(s) to be Solved by the InventionJHowever, very strict and delicate manual 
operation is required of work which contacts some specific "points" correctly with 
automata, such as a robot, and there is a problem of it being difficult to perform exact 
coordinate system setting for a short time, and being easy to come by factors, such as a 
skillful degree of an operator, out of gross errors. 

[0007] Since the tatting by automata, such as a robot, is indispensable, the restrictions of 
the ability not to make an automaton recognizing the coordinate system expressive 
medium (for example, specific reference position distant near the automaton in working 
clearance) which exists in the position outside the working range of an automaton are not 

escaped. 

[0008]Then, the purpose of the invention in this appUcation is to conquer the problem of 
the coordinate-system-setting method of such a conventional contact method. Namely, 
the invention in this application can set a coordinate system as automata, such as a robot, 
with the non-contact method using a visual sensor in the accuracy stable by easy 
operation in which the level of skill is not required of an operator. Even if a coordinate 
system expressive medium exists outside the working range of an automaton, it is in 
providing the coordinate-system-setting method which an automaton is made to 
recognize a coordinate system and can set a coordinate system as it. 
[0009] 

[Means for Solving the Problem]as a means by which this invention solves an 
aforementioned problem — "(a) — several positions which make an automaton with which 
at least one standard coordinates are set up support a camera means of a visual sensor, 
and are different, [ make take them and ] A stage of observing a coordinate system 
expressive medium which expresses a coordinate system wishing setting out visually in 
each of a position of these plurality by said camera means, and gaining sensor data about 
said coordinate system wishing setting out, (b) Based on data showing said several 
mutually different positions of an automaton, and said gained sensor data, A coordinate- 
system-setting method using a visual sensor including a stage of gaining data showing a 
position and a posture of said coordinate system wishing setting out to said at least one 
standard coordinates" is proposed. 
[0010] 

[Function]The invention in this application is replaced with the contact method by 
automata, such as the conventional robot, and the method which observes the coordinate 

system expressive medium which expressed the coordinate system visually with the 
visual sensor which made automata, such as a robot, support a camera means is used for 
it. The robot which is a typical automaton of the mold provided with the movable part 
which performs route movement hereafter according to the operation program containing 
position data is taken for an example, and the setting-out principle of the coordinate 
system in the invention in this application is explained using drawing 1 . 
[001 l]In drawing 1 . the numerals 1 express a robot and the camera 2 (for example, video 
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camera using a CCD element) which forms some visual sensors is supported by the arm 
tip. And the numerals 3 express the coordinate system expressive medium which 
expresses visually the position and posture of the coordinate system (it is called the 
following "setting-out coordinate system".) which wishes to set up. Here, it is sigmac 
about a setting-out coordinate system. ; It is referred to as Oc -Xc Yc Zc, The jig in which 
the home position Oc, Xc shaft orientations, Yc shaft orientations, and a Xc -Yc plane 
direction (Z shaft orientations turn into the direction of the outer product of the normal 
vector, i.e., the unit vector of Xc shaft orientations, and the unit vector of Yc shaft 
orientations) were expressed by the matrix form point group 4 is shown. 
[0012]The coordinate system expressive medium 3 does not necessarily need to be the jig 
in which the point group was described, and if the position and posture of a setting-out 
coordinate system are expressed visually, it can use arbitrary means. For example, the 
characterizing portions (ridgeline etc.) of the mark described in the floor line and the 
work put on the prescribed position can be used. 

[0013]Unlike conventional technology, the setting-out position (position of a coordinate 
system expressive medium) of a coordinate system does not have the restrictions which 
exist in the working range of the robot 1 that it will not become if it kicks. If the camera 2 
is a range which can put a coordinate system expressive medium into a view normally by 
making it take the suitable position for the robot 1, a coordinate system can be set as 
arbitrary positions. 

[00 14] While base coordinate system sigmab used as the standard coordinates fixed in 
working clearance is set to the robot 1 , centering on the face shield (or tool point), 
coordinate system sigmaa (a face shield coordinate system or a tool coordinate system) 
showing a robot position is set up. sigmas expresses the sensor coordinate system set as 
the visual sensor containing the camera 2. 

[0015]Here, the robot 1 is made to take the two positions Al and A2, and the position 
relationship of each coordinate system at the time of observing the coordinate system 
expressive medium 3 with the camera 2 in each robot position is considered. The 
procession (four-line four-row homogeneous transformation procession) expressing the 
relation between each coordinate system is carried out as follows. 
[0016][C]; setting out of coordinate system sigmac is performed by the data of this 
procession in procession showing the position and posture of setting-out coordinate 
system sigmac seen from base coordinate system sigmab. That is, it is a strange 
procession which should be searched for for coordinate system setting. 
[Al]; it is a procession showing the position and posture of the robot position Al seen 
from base coordinate system sigmab, and is the known procession which the robot 
recognizes. 

[0017][A2]; it is a procession showing the position and posture of the robot position A2 
seen from base coordinate system sigmab, and is the known procession which the robot 

recognizes. 

[0018][S]; it will become a known procession if the calibration of the visual sensor 
containing the camera 2 is performed in procession showing the position and posture of 
sensor coordinate system sigmas seen from the robot position. 
[0019][D1]; it is in the state which has a robot in the position Al, and is a procession 
showing the position and the posture in which setting-out coordinate system sigmac was 
seen from sensor coordinate system sigmas. It is calculated by correcting the 
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observational data based on the camera 2 of the coordinate system expressive medium 

(here point group 4) in the robot position Al using calibration data. 

[0020] [D2]; it is in the state which has a robot in the position A2, and is a procession 

showing the position and the posture in which setting-out coordinate system sigmac was 

seen from sensor coordinate system sigmas. It is calculated by correcting the 

observational data based on the camera 2 of the coordinate system expressive medium 

(here point group 4) in the robot position A2 using calibration data. 

[0021]If each procession is defined as above, both the following formulas will be 

materialized. 

[C] -- =[A1] -[S] - [Dl] ... [C] =[A2] -[S] - [D2] [Al] -\ [S], and [Dl] "^=[A2] -\ [S], and 
[D2] from (2), (1), and (2) types (3) is obtained. (1) 

[0022]Here, the robot 1 is the quantity recognized as present position data in the position 
Al and A2, and [Al] and [A2] are known. [Dl] and [D2] are [ in / are a procession 

showing the position and posture of setting-out coordinate system sigmac seen from 
sensor coordinate system sigmas in each robot position Al and A2, and / each position ] 
the characterizing portions (although it is the point group 4 which consists of two or more 
points here) on the jig 3 with the camera 2. for example, a line expresses ~ it may have ~ 
it is the quantity which observes and is calculated based on the data (interval of a point, 
etc.) about the observational data and point group 4. 

[0023]Therefore, if it asks for the data which is equivalent to [Dl] and [D2] in the two 
positions Al, A2, or a different position beyond it, a procession [C] can be searched for 
from (3) types. 

[0024]Generally the calculation which asks for the data which is equivalent to [Dl] and 
[D2] from the data about the observational data and the point group 4 which were 
acquired with the camera 2 is called the calibration of the camera. 
Various methods are known. 

Which method may be used in the invention in this application. From the point group 4 
which consists of many points to for example, two or more points (at least three pieces 
theoretically) However, when taking the aberration of a camera lens into consideration, it 
is pointed out with Mr. Tsai's model that at least seven stations are required. There is a 
method of gaining calibration data from the related pixel value data. As an example of the 
literature in which the details of the calibration of the camera used for a visual sensor 
were described, "Roger Y. Tsai, "An efficient and accurate camera calibration technique 
for 3d machine vision ", Proc. Computer Vision and. There are Pattern Recognition'86 
and pp.364-374-1986." 
[0025] 

[Example] Drawiiig 2 is the block diagram which illustrated the important section of the 
camera 2 which constitutes the visual sensor used when carrying out the invention in this 
application, and the image processing device 10 connected to this. Among a figure, the 
image processing device 10 has the central arithmetic unit (henceforth CPU) 11, and via 
the bus 21 to CPUl, The camera interface 12, the image processing processor 13, The 
console interface 4, the communication interface 15, The data memory 20 which 
comprised the TV monitor interface 16, the frame memory 17, the memory 18 for control 
soft that comprised a ROM, the program memory 19 which comprised RAM, and non- 
volatile RAM is connected. The camera 2 which put the jig 3 as a coordinate system 
expressive medium into the view in the mode of drawing 1 is connected to the camera 



Publication No. 07-311610 
Date of Publication 11.28.1995 



interface 12. 

[0026]It is connected by several sets if needed and this camera 2 detects the position of 
the work object (for example, important point working piece) performed after setting out 
of coordinate system sigmac, and it is constituted so that the data for position amendment 
may be transmitted to the robot control device (graphic display abbreviation) connected 
to the communication interface 15. 

[0027]Although the camera 2 showed only one set, it may work by connecting several 
cameras to the communication interface 15 at once. In that case, common sensor 
coordinate system sigmas is set up, for example for every camera. 
[0028]The picture caught with the camera 2 is changed into the shade image changed into 
the gray scale, and is stored in the frame memory 17. The image processing processor 13 
processes the image data stored in the frame memory 17, the image of the focus of the jig 
J is recognized, and it has the fiinction to detect the position (it is a central pixel value 
when a corresponding-picture-elements value and an image have a spread). 
[0029]The console 23 is connected to the console interface 14, and the outside console 23 
has a ten key for operating various kinds of instruction keys, input of an application 
program, edit, registration, execution, etc. outside a liquid crystal display, etc. And the 
menu for various data setting out, the list of programs, etc. can be displayed now on a 
liquid crystal display. 

[0030]The control program for CPUl 1 to control a vision sensor system is stored in the 
memory 18 for control soft. 

The program which a user produces is stored in the program memory 19. 

[003 l]The automaton (here robot) using the position information on the work etc. which 
the visual sensor detected, etc. are connected to the communication interface 15. The TV 
monitor 24 which can project selectively the picture stored in the picture or the frame 
memory 17 under photography with the camera C is connected to TV monitor interface 
16. 

[0032]Although there is no place which the above composition and function change 
fundamentally with the composition of the conventional vision sensor system, in 
operation of the invention in this application, It differs from a system conventionally in 
that it is said that a program and required data (data of the interval of the point group on 
the jig 3, the number, etc.) required in order to perform processing described into the flow 
chart of dravsj og 3 are stored in the memory 18 for control soft, or the data memory 20. 
Hereafter, the operating procedure for coordinate system sigmac setting out and 
processing are explained as what preparatory works, such as setting out of the camera 2 
and an entry of data of the jig 3, have completed. 

[0033]First, an operator operates a robot control device (graphic display abbreviation), 
and moves the robot 1 to the 1st suitable position Al to which the jig 3 is easily settled in 
the view of the camera 2 (Step SI). And the data of a procession [Al] which expresses 
the position Al from a robot control device is made to transmit, and it memorizes to the 

data memory 20 (Step S2). 

[0034]Subsequently, observation (photography) with the camera 2 is performed in the 
robot position Al, and the image of the point group 4 is captured into the frame memory 
17 (Step S3). If photography is completed, it will ask for the data of [Dl] from the pixel 
value of each point of the point group 4, and jig data (data of the interval of a point 
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group, a number, etc.) (step S4). 

[003 5] Similarly, an operator moves the robot 1 to the 2nd suitable position A2 to which 
the jig 3 is easily settled in the view of the camera 2 (Step S5), makes the data of a 
procession [A2] which expresses the position A2 from a robot control device transmit, 
and is memorized to the data memory 20 (Step S6). 

[003 6] Subsequently, observation (photography) with the camera 2 is performed in the 
robot position A2, and the image of the point group 4 is captured into the frame memory 
17 (Step S7). If photography is completed, it will ask for the data of [D2] from the pixel 
value of each point of the point group 4, and jig data (data of the interval of a point 
group, a number, etc.) (Step S8). 

[0037]By continuing step S9D, according to the above-mentioned formula (3), a 
procession [C] is calculated using these data, and a result is transmitted to a robot control 
device (Step SIO). A robot control device stores the transmitted data in a predetermined 

memory area as coordinate-system-setting data, and ends processing. 

[0038]The processing of those other than Step [ under above-mentioned procedure ] S3 

and step S7 may perform all by the robot control device side in part. 

[0039] 

[Effect of the Invention] According to the invention in this application, a coordinate 
system can be set as automata, such as a robot, with the non-contact method using a 
visual sensor in the accuracy stable by easy operation in which the level of skill is not 
required of an operator. It became possible to make an automaton recognize a coordinate 
system and to set a coordinate system as it by the invention in this application, even if a 
coordinate system expressive medium exists outside the working range of automata, such 
as a robot. 



TECHNICAL FIELD 



[Industrial ApplicationjConcerning the method of setting a coordinate system as 
automata, such as a robot, in more detail, a visual sensor is used in this invention. 
Therefore, it is related with the method of setting a coordinate system as coordinate 
system expressive media, such as a jig for coordinate system setting, by non-contact at 
automata, such as a robot. 

PRIOR ART 

[Description of the Prior Artjin the automaton of the mold provided with the movable 
part which performs route movement according to the operation program which contains 
the data (only henceforth "position data") showing a position and a posture like a robot 
and a machine tool, two or more coordinate systems are set up and the conformity 

coordinate system at the time of operation is chosen according to work contents. 
[0003 ]If the standard coordinates (henceforth a base coordinate system) fixed on the base 
of an automaton are ending with setting out, setting out of other coordinate systems will 
be attained by defining the transformation-matrix data showing the position and posture 
over the base coordinate system set [ this ] up. 

[0004]In setting up a new coordinate system with such a method conventionally, Carry 
out fixed installation of the jig which expressed the coordinate system by the point group 
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etc. to the position wishing setting out, and two or more prescribed spots of this jig are 
tagged up one by one in the portion representing the position of automata, such as a 
robot. Asking for transformation-matrix data is performed from the data of the touch 
position on a jig, and the position data of the automaton at the time of touch-up. 
[0005]The point of specifically expressing the starting point as the 1st point, for example 
(0, 0, 0), The point of expressing X shaft orientation as the 2nd point (xO, 0, 0); Point (0, 
yO, 0);yO !=0 on xO !=0 and the Y-axis which expresses the direction of an XY plane as 
the 3rd point is chosen. The pin which an automaton tends to touch is set to these three 
points, manual operation (jogging) of the automaton is carried out, and these pins are 
touched directly [ order ]. And the data (transformation-matrix data) which expresses 
relative position and posture over the base coordinate system of the coordinate system 
expressed with the jig from the position data of the automaton at the time of each tatting 
and the data of a touch position is called for. 
[0006] 



EFFECT OF THE INVENTION 



[Effect of the Invention] According to the invention in this appUcation, a coordinate 
system can be set as automata, such as a robot, with the non-contact method using a 
visual sensor in the accuracy stable by easy operation in which the level of skill is not 
required of an operator. It became possible to make an automaton recognize a coordinate 
system and to set a coordinate system as it by the invention in this application, even if a 
coordinate system expressive medium exists outside the working range of automata, such 
as a robot. 



TECHNICAL PROBLEM 



[Problem(s) to be Solved by the InvcntionJHowcvcr, very strict and delicate manual 
operation is required of work which contacts some specific "points" correctly with 
automata, such as a robot, and there is a problem of it being difficult to perform exact 
coordinate system setting for a short time, and being easy to come by factors, such as a 
skillful degree of an operator, out of gross errors. 

[0007] Since the tatting by automata, such as a robot, is indispensable, the restrictions of 
the ability not to make an automaton recognizing the coordinate system expressive 
medium (for example, specific reference position distant near the automaton in working 
clearance) which exists in the position outside the working range of an automaton are not 

escaped. 

[0008]Then, the purpose of the invention in this application is to conquer the problem of 
the coordinate-system-setting method of such a conventional contact method. Namely, 
the invention in this application can set a coordinate system as automata, such as a robot, 
with the non-contact method using a visual sensor in the accuracy stable by easy 
operation in which the level of skill is not required of an operator. Even if a coordinate 
system expressive medium exists outside the working range of an automaton, it is in 
providing the coordinate-system-setting method which an automaton is made to 
recognize a coordinate system and can set a coordinate system as it. 
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MEANS 



[Means for Solving the Problemjas a means by which this invention solves an 
aforementioned problem ~ "(a) ~ several positions which make an automaton with which 
at least one standard coordinates are set up support a camera means of a visual sensor, 
and are different, [ make take them and ] A stage of observing a coordinate system 
expressive medium which expresses a coordinate system wishing setting out visually in 
each of a position of these plurality by said camera means, and gaining sensor data about 
said coordinate system wishing setting out, (b) Based on data showing said several 
mutually different positions of an automaton, and said gained sensor data, A coordinate- 
system-setting method using a visual sensor including a stage of gaining data showing a 
position and a posture of said coordinate system wishing setting out to said at least one 
standard coordinates" is proposed. 



OPERATION 



[FunctionJThe invention in this application is replaced with the contact method by 
automata, such as the conventional robot, and the method which observes the coordinate 
system expressive medium which expressed the coordinate system visually with the 
visual sensor which made automata, such as a robot, support a camera means is used for 
it. The robot which is a typical automaton of the mold provided with the movable part 
which performs route movement hereafter according to the operation program containing 
position data is taken for an example, and the setting-out principle of the coordinate 
system in the invention in this application is explained using drawing 1 . 
[001 l]In dramng 1 . the numerals 1 express a robot and the camera 2 (for example, video 
camera using a CCD element) which forms some visual sensors is supported by the arm 
tip. And the numerals 3 express the coordinate system expressive medium which 
expresses visually the position and posture of the coordinate system (it is called the 
following "setting-out coordinate system".) which wishes to set up. Here, it is sigmac 
about a setting-out coordinate system. ; It is referred to as Oc -Xc Yc Zc, The jig in which 
the home position Oc, Xc shaft orientations, Yc shaft orientations, and a Xc -Yc plane 
direction (Z shaft orientations turn into the direction of the outer product of the normal 
vector, i.e., the unit vector of Xc shaft orientations, and the unit vector of Yc shaft 
orientations) were expressed by the matrix form point group 4 is shown. 
[0012]The coordinate system expressive medium 3 does not necessarily need to be the jig 
in which the point group was described, and if the position and posture of a setting-out 
coordinate system are expressed visually, it can use arbitrary means. For example, the 
characterizing portions (ridgeline etc.) of the mark described in the floor line and the 
work put on the prescribed position can be used. 

[0013]Unlike conventional technology, the setting-out position (position of a coordinate 
system expressive medium) of a coordinate system does not have the restrictions which 
exist in the working range of the robot 1 that it will not become if it kicks, If the camera 2 
is a range which can put a coordinate system expressive medium into a view normally by 
making it take the suitable position for the robot 1, a coordinate system can be set as 
arbitrary positions. 
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[00 14] While base coordinate system sigmab used as the standard coordinates fixed in 
working clearance is set to the robot 1 , centering on the face shield (or tool point), 
coordinate system sigmaa (a face shield coordinate system or a tool coordinate system) 
showing a robot position is set up. sigmas expresses the sensor coordinate system set as 
the visual sensor containing the camera 2. 

[0015]Here, the robot 1 is made to take the two positions Al and A2, and the position 
relationship of each coordinate system at the time of observing the coordinate system 
expressive medium 3 with the camera 2 in each robot position is considered. The 
procession (four-line four-row homogeneous transformation procession) expressing the 
relation between each coordinate system is carried out as follows. 
[0016][C]; setting out of coordinate system sigmac is performed by the data of this 
procession in procession showing the position and posture of setting-out coordinate 
system sigmac seen from base coordinate system sigmab. That is, it is a strange 
procession which should be searched for for coordinate system setting. 
[Al]; it is a procession showing the position and posture of the robot position Al seen 
from base coordinate system sigmab, and is the known procession which the robot 
recognizes. 

[0017][A2]; it is a procession showing the position and posture of the robot position A2 
seen from base coordinate system sigmab, and is the known procession which the robot 

recognizes. 

[0018][S]; it will become a known procession if the calibration of the visual sensor 

containing the camera 2 is performed in procession showing the position and posture of 

sensor coordinate system sigmas seen from the robot position. 

[0019][D1]; it is in the state which has a robot in the position Al, and is a procession 

showing the position and the posture in which setting-out coordinate system sigmac was 

seen from sensor coordinate system sigmas. It is calculated by correcting the 

observational data based on the camera 2 of the coordinate system expressive medium 

(here point group 4) in the robot position Al using calibration data. 

[0020] [D2]; it is in the state which has a robot in the position A2, and is a procession 

showing the position and the posture in which setting-out coordinate system sigmac was 

seen from sensor coordinate system sigmas. It is calculated by correcting the 

observational data based on the camera 2 of the coordinate system expressive medium 

(here point group 4) in the robot position A2 using calibration data. 

[0021]If each procession is defined as above, both the following formulas will be 

materialized. 

[C] - =[A1] -[S] - [Dl] ... [C] =[A2] -[S] - [D2] [Al] -\ [S], and [Dl] MA2] -\ [S], and 
[D2] from (2), (1), and (2) types (3) is obtained. (1) 

[0022]Here, the robot 1 is the quantity recognized as present position data in the position 
Al and A2, and [Al] and [A2] are known. [Dl] and [D2] are [ in / are a procession 
showing the position and posture of setting-out coordinate system sigmac seen from 
sensor coordinate system sigmas in each robot position Al and A2, and / each position ] 
the characterizing portions (although it is the point group 4 which consists of two or more 
points here) on the jig 3 with the camera 2. for example, a line expresses ~ it may have ~ 
it is the quantity which observes and is calculated based on the data (interval of a point, 
etc.) about the observational data and point group 4. 

[0023]Therefore, if it asks for the data which is equivalent to [Dl] and [D2] in the two 
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positions Al, A2, or a different position beyond it, a procession [C] can be searched for 
from (3) types. 

[0024]GeneraIIy the calculation which asks for the data which is equivalent to [Dl] and 
[D2] from the data about the observational data and the point group 4 which were 
acquired with the camera 2 is called the calibration of the camera. 
Various methods are known. 

Which method may be used in the invention in this application. From the point group 4 
which consists of many points to for example, two or more points (at least three pieces 
theoretically) However, when taking the aberration of a camera lens into consideration, it 
is pointed out with Mr. Tsai's model that at least seven stations are required. There is a 
method of gaining calibration data from the related pixel value data. As an example of the 
literature in which the details of the calibration of the camera used for a visual sensor 
were described, "Roger Y. Tsai, "An efficient and accurate camera calibration technique 
for 3d machine vision ", Proc. Computer Vision and. There are Pattern Recognition'86 
and pp.364-374- 1986." 



EXAMPLE 



[Hxamplc]Dravving 2 is the block diagram which illusfrated the important section of the 
camera 2 which constitutes the visual sensor used when carrying out the invention in this 
application, and the image processing device 10 connected to this. Among a figure, the 
image processing device 10 has the central arithmetic unit (henceforth CPU) 11, and via 
the bus 2 1 to CPU 1 , The camera interface 12, the image processing processor 13, The 
console interface 4, the communication interface 15, The data memory 20 which 
comprised the TV monitor interface 16, the frame memory 17, the memory 18 for confrol 
soft that comprised a ROM, the program memory 19 which comprised RAM, and non- 
volatile RAM is connected. The camera 2 which put the jig 3 as a coordinate system 
expressive medium into the view in the mode of drawing 1 is connected to the camera 
interface 12. 

[0026]It is connected by several sets if needed and this camera 2 detects the position of 
the work object (for example, important point working piece) performed after setting out 
of coordinate system sigmac, and it is constituted so that the data for position amendment 
may be fransmitted to the robot confrol device (graphic display abbreviation) connected 
to the communication interface 15. 

[0027]Although the camera 2 showed only one set, it may work by connecting several 
cameras to the communication interface 15 at once. In that case, common sensor 
coordinate system sigmas is set up, for example for every camera. 
[0028]The picture caught with the camera 2 is changed into the shade image changed into 
the gray scale, and is stored in the frame memory 17. The image processing processor 13 
processes the image data stored in the frame memory 17, the image of the focus of the jig 
J is recognized, and it has the function to detect the position (it is a central pixel value 
when a corresponding-picture-elements value and an image have a spread). 
[0029]The console 23 is connected to the console interface 14, and the outside console 23 
has a ten key for operating various kinds of instruction keys, input of an application 
program, edit, regisfration, execution, etc. outside a liquid crystal display, etc. And the 
menu for various data setting out, the list of programs, etc. can be displayed now on a 
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liquid crystal display. 

[0030]The control program for CPUl 1 to control a vision sensor system is stored in the 
memory 1 8 for control soft. 

The program which a user produces is stored in the program memory 19. 

[0031]The automaton (here robot) using the position information on the work etc. which 
the visual sensor detected, etc. are connected to the communication interface 15. The TV 
monitor 24 which can project selectively the picture stored in the picture or the frame 
memory 17 under photography with the camera C is connected to TV monitor interface 
16. 

[0032]Although there is no place which the above composition and function change 
fundamentally with the composition of the conventional vision sensor system, in 
operation of the invention in this application, It differs fi-om a system conventionally in 
that it is said that a program and required data (data of the interval of the point group on 
the jig 3, the number, etc.) required in order to perform processing described into the flow 
chart of drawing 3 are stored in the memory 18 for control soft, or the data memory 20. 
Hereafter, the operating procedure for coordinate system sigmac setting out and 
processing are explained as what preparatory works, such as setting out of the camera 2 
and an entry of data of the jig 3, have completed. 

[0033]First, an operator operates a robot control device (graphic display abbreviation), 
and moves the robot 1 to the 1st suitable position Al to which the jig 3 is easily settled in 
the view of the camera 2 (Step SI). And the data of a procession [Al] which expresses 
the position Al from a robot control device is made to transmit, and it memorizes to the 
data memory 20 (Step S2). 

[0034]Subsequently, observation (photography) with the camera 2 is performed in the 
robot position Al , and the image of the point group 4 is captured into the frame memory 
17 (Step S3). If photography is completed, it will ask for the data of [Dl] from the pixel 
value of each point of the point group 4, and jig data (data of the interval of a point 
group, a number, etc.) (step S4). 

[003 5] Similarly, an operator moves the robot 1 to the 2nd suitable position A2 to which 
the jig 3 is easily settled in the view of the camera 2 (Step S5), makes the data of a 
procession [A2] which expresses the position A2 from a robot control device transmit, 
and is memorized to the data memory 20 (Step S6). 

[003 6] Subsequently, observation (photography) with the camera 2 is performed in the 
robot position A2, and the image of the point group 4 is captured into the frame memory 
17 (Step S7). If photography is completed, it will ask for the data of [D2] from the pixel 
value of each point of the point group 4, and jig data (data of the interval of a point 
group, a number, etc.) (Step SB). 

[0037]By continuing step S9D, according to the above-mentioned formula (3), a 
procession [C] is calculated using these data, and a result is transmitted to a robot confrol 
device (Step SIO). A robot control device stores the transmitted data in a predetermined 
memory area as coordinate-system-setting data, and ends processing. 
[0038]The processing of those other than Step [ under above-mentioned procedure ] S3 
and step S7 may perform all by the robot confrol device side in part. 
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DESCRIPTION OF DRAWINGS 



[Brief Description of the Drawings] 

LQlf];:^J.i.\g..l.l't •J' figure for taking a robot for an example and explaining the setting-out 
principle of the coordinate system in the invention in this application. 
[Drawing 2 | It is the block diagram which illustrated the important section of the camera 
which constitutes the visual sensor used when carrying out the invention in this 
application, and the image processing device connected to this. 

[ Drawin g 3] It is a flow chart explaining the operation for setting up a coordinate system 
according to the invention in this application, and procedure. 
[Description of Notations] 

1 Robot 

2 Camera 

3 Coordinate system expressive medium (jig) 

4 Point group 

10 Image processing device 

1 1 Central arithmetic unit (CPU) 

12 Camera interface 

13 Image processing processor 

14 Console interface 

15 Communication interface 

16 TV monitor interface 

17 Frame memory 

1 8 The memory for control soft 

19 Program memory 

20 Data memory 

21 Bus 

23 Console 

24 TV monitor 
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[Drawing 3] 
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